Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kprom: Multiple clients #1

Merged
merged 1 commit into from
Sep 17, 2024
Merged

kprom: Multiple clients #1

merged 1 commit into from
Sep 17, 2024

Conversation

lukasrynt
Copy link
Owner

@lukasrynt lukasrynt commented Sep 13, 2024 •
edited
Loading

Issue

The main goal of this PR is to add the ability for kprom plugin to differentiate between multiple clients.

The current implementation of the WithClientLabel() option in the kprom package assigns metrics to the most recently registered kgo client, which leads to metrics being grouped under a static client_id. The goal is to keep the existing API but make the client_id label dynamic, allowing it to reflect the actual client making the request rather than being statically set at first registration (see twmb#815).

Only a subset of metrics will be affected by this change:

  • write_bytes_total
  • write_errors_total
  • write_wait_seconds
  • write_time_seconds
  • read_bytes_total
  • read_errors_total
  • read_time_seconds
  • request_duration_e2e_seconds
  • request_throttled_seconds
  • produce_compressed_bytes_total
  • produce_(uncompressed_)bytes_total
  • produce_batches_total
  • produce_records_total
  • fetch_compressed_bytes_total
  • fetch_(uncompressed_)bytes_total
  • fetch_batches_total
  • fetch_records_total

Metrics that will continue to have a constant client_id:

  • connects_total
  • connect_errors_total
  • disconnects_total
  • buffered_fetch_records_total
  • buffered_produce_records_total

Testing

I tested the updated metrics locally using a combination of consumers and producers. When scraping Prometheus, the metrics are now properly separated by the client_id label, even when writing to and reading from the same topic using different client IDs for two producers and consumers:

# HELP test_buffered_fetch_records_total Total number of records buffered within the client ready to be consumed
# TYPE test_buffered_fetch_records_total gauge
test_buffered_fetch_records_total{client_id="primary"} 0
# HELP test_buffered_produce_records_total Total number of records buffered within the client ready to be produced
# TYPE test_buffered_produce_records_total gauge
test_buffered_produce_records_total{client_id="primary"} 0
# HELP test_connects_total Total number of connections opened
# TYPE test_connects_total counter
test_connects_total{client_id="primary",node_id="1"} 8
test_connects_total{client_id="primary",node_id="seed_0"} 4
# HELP test_fetch_bytes_total Total number of uncompressed bytes fetched
# TYPE test_fetch_bytes_total counter
test_fetch_bytes_total{client_id="primary",node_id="1",topic="primary"} 800
test_fetch_bytes_total{client_id="secondary",node_id="1",topic="primary"} 800
# HELP test_produce_bytes_total Total number of uncompressed bytes produced
# TYPE test_produce_bytes_total counter
test_produce_bytes_total{client_id="primary",node_id="1",topic="primary"} 400
test_produce_bytes_total{client_id="secondary",node_id="1",topic="primary"} 400
# HELP test_read_bytes_total Total number of bytes read
# TYPE test_read_bytes_total counter
test_read_bytes_total{client_id="primary",node_id="1"} 14029
test_read_bytes_total{client_id="primary",node_id="seed_0"} 1166
test_read_bytes_total{client_id="secondary",node_id="1"} 13876
test_read_bytes_total{client_id="secondary",node_id="seed_0"} 1166
# HELP test_write_bytes_total Total number of bytes written
# TYPE test_write_bytes_total counter
test_write_bytes_total{client_id="primary",node_id="1"} 10080
test_write_bytes_total{client_id="primary",node_id="seed_0"} 172
test_write_bytes_total{client_id="secondary",node_id="1"} 10072
test_write_bytes_total{client_id="secondary",node_id="seed_0"} 180

Implementation

Since kgo doesn’t export client_id in a way that supports dynamic labeling in hooks, I had to make some modifications to the core library. My goal was to minimize changes to the kgo package itself and to avoid breaking any existing APIs (such as hooks). To that end, I chose not to add dynamic labels to the connection and buffered metrics, keeping those labels static.

@lukasrynt lukasrynt force-pushed the kprom/dynamic-client-id-label branch 3 times, most recently from d33699b to 599b103 Compare September 16, 2024 09:30
@lukasrynt lukasrynt force-pushed the kprom/dynamic-client-id-label branch from 599b103 to 0ea78bc Compare September 16, 2024 09:35
@lukasrynt lukasrynt changed the title Support metrics separated by dynamic client_id label kprom: Multiple clients Sep 17, 2024
@lukasrynt lukasrynt merged commit 7f8d2d4 into master Sep 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@lukasrynt